Rate-distortion (RD) theory is a fundamental theory for compression. Rate distortion based techniques have started to be exploited in practical video coding systems in recent years. Usually the constrained optimization problem is transferred into unconstrained problem through a lagrangian method. Therefore, the selection of a lagrangian multiplier becomes a critical issue relating to video quality.
According to information theory, the lagrangian multiplier corresponds to the negative slope of the rate distortion (R-D) curve of the input signal processed with the coding system. In practice, both the signal characteristics and the coding system settings will affect this operational R-D behavior. On the one hand, video signals are complex and dynamic. No single mathematical model can describe all the scenes, and different signal characteristics, such as spatial frequency and temporal motion will lead to different R-D behaviors. Meanwhile, the human visual system (HVS) is more sensitive to some feature areas than to others. The lagrangian multiplier should be selected adaptively to those input signal features and to the characteristics of the human visual system. On the other hand, the R-D curve is the operational behavior of a particular coding system, rather than a theoretical boundary. The parameter settings of the coding control will affect the rate distortion behavior of the input signal. For example, the same content coded under intra mode and inter mode will result in different R-D curves.
Conventional R-D based techniques do not take into account input signal characteristics (i.e., all input signals are treated equally). Furthermore, coding parameters considered for the R-D curve are not sufficient, since only some simple assumptions between the lagrangian multiplier and coding schemes such as picture type, and/or reference picture distance have been considered.
It would be desirable to implement a rate distortion optimization system that adapts to features of varying video signals as well as features of different coding parameters.